196-2007: Pipes and Threads: Performance Testing of Advanced Scalability Features in SAS®9

SAS v9 provides many new features of interest to users with large data sets. Many of the commonly used procedures are multi-treaded. This allows users to harness the full power of multi-processor architectures to enhance the efficiency of some tasks. In addition, SAS/Connect allows users to spawn multiple independent processes to execute custom parallel processing solutions. SAS/Connect also enables pipeline parallelism, i.e., piping of records from one DATA/PROC step to another between processes. This potentially relieves some of the I/O burden of the total SAS job, an important consideration when dealing with large data sets. We examined the real world speed of these three features using two different multiprocessor system architectures, a Sun Microsystems Sun Fire 3800 running Unix and an IBM 9672/Y36 running MVS. We investigated a typical usage pattern with a simple DATA step followed by a PROC TABULATE. Test data were from a simulated insurance claims file with test runs ranging from 10 million to approximately 322 million records. Our tests suggest that for an architecture based on traditional synchronous processing using the traditional method of the DATA step writing intermediate output to disk, SAS v9 is not substantially different than SAS v8. However, the enhanced features in SAS v9, piping and multi-threading, yield significantly shorter total run times on more powerful architectures. These performance gains were evident at all file sizes tested (1.2 to 38.4 GB). We noted also that the built-in multi-threading in PROC TABULATE performed better than a manual parallel processing solution. This is an important finding since it suggests that shorter run times can be achieved directly with the base product without the added expense of an additional complicated programming effort. INTRODUCTION The introduction of SAS v9 marked a major upgrade in the SAS computing system with many features targeting scalability for large datasets. Many procedures have been enhanced with the ability to perform parallel processing. This includes some of the most frequently used procedures (MEANS, REPORT, SORT, SQL, SUMMARY, TABULATE, GLM, REG). Introduced in V8, SAS/CONNECT gives you the ability to exploit SMP (Symmetric Multi-Processing) hardware as well as network resources to perform parallel processing and easily coordinate all the results into a single client SAS session. In v9, SAS/CONNECT supports pipeline parallelism, which allows multiple DATA steps or procedures to execute in parallel and to pipe the output from one process as the input to the next process in a pipeline. Piping improves performance and reduces the demand for disk space. SYSTEM ARCHITECTURES In our testing we used two basic architectures.